In recent years, there have been known techniques of making expressions in which a real space and a virtual space are fused, by using a stereoscopic technique. One example of such techniques superimposes CG images of a model for CG shot by left and right virtual cameras, on real images shot by left and right imaging sections, thereby displaying a stereoscopic image in which a real space and a virtual space are fused.
However, in the above example of the techniques, the CG images are generated by using an image different from an image obtained by shooting a real space. Therefore, this technique is insufficient for making various expressions by fusing the CG images and the image of a real space. That is, the CG images, which are superimposed on an image obtained by shooting a real space, are images obtained by shooting a model prepared in advance with a virtual camera. Therefore, a manner of displaying the model has no diversity, and the range of expression is limited.
Therefore, a feature of the illustrative embodiments is to provide an information processing technique capable of making various expressions by using an image shot by a real camera, and a virtual model.
The illustrative embodiments employ the following configurations to solve the above problems.
One aspect of the illustrative embodiments is a computer-readable storage medium having stored therein an information processing program executed by a computer included in an information processing apparatus connected to a left real camera and a right real camera for shooting a real space, and to display means capable of displaying a stereoscopic visible image, the information processing program causing a computer to function as: real image obtaining means; virtual camera setting means; left texture area setting means; right texture area setting means; left virtual camera image obtaining means; right virtual camera image obtaining means; and display control means. The real image obtaining means obtains a real image for a left eye shot by the left real camera, and a real image for a right eye shot by the right real camera. The virtual camera setting means sets a left virtual camera and a right virtual camera in a virtual space such that the left virtual camera and the right virtual camera are placed in accordance with a position and an orientation of the left real camera in the real space, and a position and an orientation of the right real camera in the real space. The left texture area setting means sets, as a left texture area, an area in the real image for a left eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the left virtual camera set by the virtual camera setting means. The right texture area setting means sets, as a right texture area, an area in the real image for a right eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the right virtual camera set by the virtual camera setting means. The left virtual camera image obtaining means applies an image included in the left texture area set by the left texture area setting means, to a drawing model that is the same as the virtual model or has a predetermined relationship with the virtual camera, and obtains a left virtual camera image of the drawing model shot by the left virtual camera. The right virtual camera image obtaining means applies an image included in the right texture area set by the right texture area setting means, to the drawing model, and obtains a right virtual camera image of the drawing model shot by the right virtual camera. The display control means displays the left virtual camera image and the right virtual camera image on the display means such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.
The virtual camera setting means may set the left virtual camera, based on relative positions and relative orientations of a predetermined subject present in the real space and the real camera for a left eye with respect to each other. That is, the virtual camera setting means may set the left virtual camera, based on the position and orientation of the real camera for a left eye relative to a predetermined subject present in the real space, or based on the position and orientation of a predetermined subject present in the real space relative to the real camera for a left eye. Alternatively, the virtual camera setting means may set the left virtual camera, based on the absolute position and the absolute orientation of the left real camera detected by means for detecting an absolute position (for example, a GPS), and orientation detection means (for example, an angular velocity sensor, an acceleration sensor, or means for detecting a geomagnetism). In the same manner, the virtual camera setting means may set the right virtual camera, based on relative positions and relative orientations of a predetermined subject present in the real space and the real camera for a right eye with respect to each other, or based on the absolute position and the absolute orientation of the right real camera.
According to the above configuration, it is possible to apply each of the real image for a left eye and the real image for a right eye as textures to the virtual model, and to stereoscopically display the virtual model. Also, it is possible to vary a display manner of the virtual model depending on the obtained real images. In addition, since each of the real image for a left eye and the real image for a right eye is used as textures, it is possible to generate the left virtual camera image and the right virtual camera image without reducing the resolution in comparison with the case where one of the real images is used.
In another aspect of the illustrative embodiments, the left texture area setting means may set, as a left texture area, an area corresponding to the position and the contour of the virtual model in a rendered image obtained by shooting the virtual model by the left virtual camera. The right texture area setting means may set, as a right texture area, an area corresponding to the position and the contour of the virtual model in a rendered image obtained by shooting the virtual model by the right virtual camera.
According to the above configuration, it is possible to set, as a texture area, an area corresponding to the position and the contour of the virtual model in an image obtained by shooting the virtual model by each virtual camera.
In another aspect of the illustrative embodiments, the information processing program may further cause the computer to function as superimposed image generation means for generating a superimposed image for a left eye obtained by superimposing the left virtual camera image onto the real image for a left eye, and a superimposed image for a right eye obtained by superimposing the right virtual camera image onto the real image for a right eye. Then, the display control means displays the superimposed image for a left eye and the superimposed image for a right eye on the display means.
According to the above configuration, it is possible to display a superimposed image including the real image for a left eye, and the left virtual camera image obtained by shooting the virtual model, and a superimposed image including the real image for a right eye, and the right virtual camera image obtained by shooting the virtual model.
In another aspect of the illustrative embodiments, the information processing program may further cause the computer to function as deformation object setting means for generating a deformation object into which the virtual model is deformed, and setting the deformation object in the virtual space. The left virtual camera image obtaining means applies an image included in the left texture area set by the left texture area setting means, to the deformation object, and obtains the left virtual camera image of the deformation object shot by the left virtual camera. The right virtual camera image obtaining means applies an image included in the right texture area set by the right texture area setting means, to the deformation object, and obtains the right virtual camera image of the deformation object shot by the right virtual camera.
According to the above configuration, it is possible to 3-dimensionally vary a shot image of the real space. For example, by deforming a virtual model on a plane, an image including a plane, in the real world, that has been deformed can be displayed.
In another aspect of the illustrative embodiments, the virtual model may have a planar shape or a curved-surface shape, and may be set in the virtual space so as to follow the shape of a plane or a curved surface in the real space. The deformation object setting means generates the deformation object into which the virtual model having a planar shape or a curved-surface shape, and sets the deformation object in the virtual space.
According to the above configuration, it is possible to display a scene in which a plane or a curved surface in the real space varies 3-dimensionally.
In another aspect of the illustrative embodiments, the deformation object setting means may generate the deformation object into which the virtual model is deformed without its end portion being deformed.
According to the above configuration, when the virtual camera image is superimposed onto the real image, the resultant image does not provide a feeling of strangeness at the boundary of the deformation object.
In another aspect of the illustrative embodiments, the information processing program may further cause the computer to function as second object setting means for setting a second object different from the deformation object in the virtual space such that the second object has a predetermined position relationship with the deformation object. The left virtual camera image obtaining means obtains a left virtual camera image including the deformation object and the second object shot by the left virtual camera. The right virtual camera image obtaining means obtains a right virtual camera image of the deformation object and the second object shot by the right virtual camera.
According to the above configuration, the second object is set in the virtual space so as to have a predetermined position relationship with the deformation object, and an image obtained by superimposing an image of the second object in the virtual space onto an image of the real space that has been deformed, is displayed. Therefore, the second object appears to actually exist having the predetermined position relationship with the deformed portion in the real space. For example, it is possible to display a water surface that is heaving as a result of deformation of a plane in the real space, and a fish present in water.
In another aspect of the illustrative embodiments, the second object may be placed so as to be away from the deformation object in the shooting direction of each of the left virtual camera and the right virtual camera. The information processing program may further cause the computer to function as mask object setting means for setting a predetermined mask object for masking the second object, at an end portion of the deformation object.
According to the above configuration, when the virtual camera is set at a position where the second object is not visible via the deformation object (when the second object is shot from an oblique direction), the second object is masked.
In another aspect of the illustrative embodiments, the left real camera and the right real camera may shoot a marker, having a predetermined planar shape, that is placed in the real space. The virtual camera setting means may set the left virtual camera and the right virtual camera in the virtual space such that the left virtual camera and the right virtual camera are placed in accordance with the position and the orientation of the left real camera in the real space relative to the position and the orientation of the marker, and the position and the orientation of the right real camera in the real space relative to the position and the orientation of the marker. The virtual model may be a model, having a planar shape, that is placed on a plane on which the marker is placed, in the marker coordinate system set based on the marker. In addition, the information processing program may further cause the computer to function as deformation object setting means for generating the deformation object into which the virtual model is deformed, and setting the deformation object in the virtual space. The left virtual camera image obtaining means applies, as a texture, an image included in the left texture area set by the left texture area setting means, to the deformation object, and obtains a left virtual camera image of the deformation object shot by the left virtual camera. The right virtual camera image obtaining means applies, as a texture, an image included in the right texture area set by the right texture area setting means, to the deformation object, and obtains a right virtual camera image of the deformation object shot by the right virtual camera.
According to the above configuration, it is possible to stereoscopically display a scene in which a plane in the real space on which the marker is placed varies. As a result, for example, it is possible to display a scene in which the plane in the real world is heaving as a water surface.
In another aspect of the illustrative embodiments, the virtual camera setting means may detect a specific subject included in the real image for a left eye, obtain relative positions and relative orientations of the left real camera and the specific subject present in the real space with respect to each other, based on a result of the detection, and set the left virtual camera, based on the relative positions and the relative orientations. In addition, the virtual camera setting means may detect the specific subject included in the real image for a right eye, obtain relative positions and relative orientations of the right real camera and the specific subject present in the real space with respect to each other, based on a result of the detection, and set the right virtual camera, based on the relative positions and the relative orientations.
According to the above configuration, by shooting the specific subject, it is possible to place the left virtual camera and the right virtual camera such that the positions and the orientations of the left virtual camera and the right virtual camera correspond to the positions and the orientations of the left real camera and the right real camera in the real space, without using another component in addition to the left real camera and the right real camera.
In another aspect of the illustrative embodiments, the virtual model may be placed based on a position, in the virtual space, that corresponds to the position of the specific subject in the real space.
According to the above configuration, since the virtual model is placed at the position of the specific object, if the specific subject is shot, the virtual model can always be shot by the virtual camera.
The illustrative embodiments may be an information processing apparatus that realizes the above means. Alternatively, the illustrative embodiments may be an information processing system including a plurality of components that realize the above means and operate in a coordinated manner. The information processing system may include one apparatus, or may include a plurality of apparatuses.
According to the illustrative embodiments, it is possible to realize various expressions by using an image shot by a real camera, and a virtual model.
These and other objects, features, aspects and advantages of the illustrative embodiments will become more apparent from the following detailed description of the illustrative embodiments when taken in conjunction with the accompanying drawings.